When and how convolutional neural networks generalize to out-of-distribution category–viewpoint combinations

Object recognition and viewpoint estimation lie at the heart of visual understanding. Recent studies have suggested that convolutional neural networks (CNNs) fail to generalize out-of-distribution (OOD) category–viewpoint combinations, is, combinations not seen during training. Here we investigate when how such OOD generalization may be possible by evaluating CNNs trained classify both object category three-dimensional on identifying mechanisms facilitate generalization. We show increasing number in-distribution (data diversity) substantially improves even with same amount training data. compare learning in separate shared network architectures, observe starkly different trends while are helpful distribution, significantly outperform ones combinations. Finally, demonstrate is facilitated mechanism specialization, emergence two types neuron—neurons selective invariant viewpoint, vice versa. The combination essential for However, often were authors impact data diversity architectural choices capability generalizing